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After 12 years focused on developing school leaders who act as change agents for educational equity, the 
Principal Residency Network (PRN) partnered with Johnson and Wales University’s Center for Research and 
Evaluation to conduct a utilization-focused (Patton, 2002) program evaluation funded by a grant from the Rhode 
Island Foundation. The PRN is a principal preparation program of the non-profit organization, the Center for 
Leadership and Educational Equity. This sequential explanatory mixed methods study explored PRN graduates’ 
outcomes and perceptions of the program, with an overarching purpose of creating a coherent data collection 
and inquiry process to be used by program staff on an ongoing basis. Following the development of an 
evaluation framework, Phase I of the study consisted of collecting assessment data and feedback from current 
PRN participants, as well as administering a survey questionnaire to recent graduates of the program (N=14), 
previously administered in 2005 (N=21) and 2009 (N=6). Phase II of this evaluation was designed to further 
explore recent graduates’ perceptions of the nature and relevancy of the program in developing their 
commitment and skill to lead for equity in order to recommend program improvements; N=7 participated in a 90 
minute focus group. Findings indicated four conclusions from which recommendations were drawn: the program 
is achieving strong results, participants perceive the program to have an interconnected and coherent focus on 
preparing them to be equity-oriented leaders, the mentor is a critical component, and modeling the cycle of 
inquiry created through this evaluative study is important. 

PROGRAM FRAMEWORK 

The Principal Residency Network (PRN) is a principal preparation program of the Center for 
Leadership and Educational Equity. The program was initiated in 2000 as a state-approved 
administrator certification program featuring an intensive residency with a mentor principal 
and a cohort structure. The PRN has continuously identified, implemented, and refined the 
research-based practices identified in Table 1 through ongoing efforts to collect and evaluate 
data for the purpose of program improvement. For a more extensive discussion of the 
theoretical framework and literature that supports the principal preparation practices listed in 
Table 1 and used by the PRN, see Braun, Gable, & Kite (201 la; 201 lb). 
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Table 1 

Principal Preparation Program Practices and Supporting Literature 


Practices _ 

Structural 

• Partnerships between 
universities and districts 

• Program developers’ 
commitment 

• Rigorous entrance 
requirements for strong 
and diverse candidates 

• Financial support, release 
time for participants 

• Supportive district and 
state infrastructure 

• Program monitoring for 
improvement 

Content 

• Standards-based content 

• Coherent and relevant 
curriculum 

• Individualized content 

• Focus on shared 
instructional leadership 

• Focus on equity and 

school reform _ 

Delivery 

• High quality internship 


• Problem-based learning 


• Mentoring or coaching 


• Cohort structure 


Supporting Research and Reviews of Literature 


Darling-Hammond, LaPointe, Meyerson, Orr, & Cohen, 2007; 
Jackson & Kelly, 2002; Milstein & Krueger, 1997; Murphy, 

1993, 1999; Orr, 2006; SREB, 2006; USDOE, 2004 
Darling-Hammond et al., 2007; USDOE, 2004 

Bredeson, 1996; Darling-Hammond et al., 2007; Hart & Pounder, 
1999; Jackson & Kelly, 2002; Lauder, 2000; Leithwood & Jantzi, 
1996; Milstein & Krueger, 1997; Murphy, 1993 ; Orr, 2006; 
SREB, 2006; USDOE, 2004 

Darling-Hammond et al., 2007; Leithwood & Jantzi, 1996; 
Milstein & Krueger, 1997; SREB, 2006 
Darling-Hammond et al., 2007; Orr, 2006; SREB, 2006 

Lauder, 2000; Leithwood & Jantzi, 1996; Milstein & Krueger, 
1997; Orr, 2006; SREB, 2006; USDOE, 2004 

Darling-Hammond et al., 2007; Jackson & Kelly, 2002; Lauder, 
2000; Orr, 2006; SREB, 2006; USDOE, 2004 
Darling-Hammond et al., 2007; Jackson & Kelly, 2002; Milstein 
& Krueger, 1997; Orr, 2006; SREB, 2006; USDOE, 2003 
Jackson & Kelly, 2002; Lauder, 2000; Leithwood & Jantzi, 1996 
Elmore, 1999; Jackson & Kelly, 2002; LaPoint, Meyerson, & 
Darling-Hammond, 2005; Leithwood & Jantzi; 1996; McCarthy, 
1999; Orr, 2006; SREB, 2006 

Jackson & Kelly, 2002; LaPoint et al., 2005; Leithwood & Jantzi, 
1996; McCarthy, 1999; Murphy, 1999; Orr, 2006; SREB, 2006 


Darling-Hammond et al., 2007; Hart & Pounder, 1999; Jackson & 
Kelly, 2002; Lauder, 2000; Leithwood & Jantzi, 1996; 

Leithwood, Seashore Louis, Anderson, & Wahlstron, 2004; 
Murphy, 1993; SREB, 2006; USDOE, 2004 
Darling-Hammond et al., 2007; Hart & Pounder, 1999; Jackson & 
Kelly, 2002; Lauder, 2000; Leithwood & Jantzi, 1996; 

Leithwood et al., 2004; McCarthy, 1999; Murphy, 1993, 1999; 
Orr, 2006; SREB, 2006 

Darling-Hammond et al., 2007; Jackson & Kelly, 2002; Lauder, 
2000; Leithwood & Jantzi, 1996; Milstein & Krueger, 1997; 
Murphy, 1993; SREB, 2006 

Darling-Hammond et al., 2007; Hart & Pounder, 1999; Jackson & 
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Kelly, 2002; Leithwood & Jantzi, 1996; McCarthy, 1999; 
Milstein & Krueger, 1997; USDOE, 2004 

• Habit of Reflection Davis, Darling-Hammond, LaPoint, & Meyerson, 2005; LaPoint 

et al., 2005; Lauder, 2000; Leithwood & Jantzi, 1996; Milstein & 
Krueger, 1997; SREB, 2006 

• Performance assessments Hart & Pounder, 1999; Jackson & Kelly, 2002; Lauder, 2000; 

Leithwood & Jantzi, 1996; Orr, 2006; SREB, 2006; USDOE, 
2004 


With a mission to develop principals who champion educational change through 
leadership of innovative schools for the purpose of improving student achievement, the 
program has a significant focus on preparing administrators to lead for equity. This approach 
is grounded in the assumption that educational leaders can increase equitable outcomes for all 
students in schools through specific practices (Ross & Berger, 2009). The equity-oriented 
leadership practices the PRN aims to enable school leaders to enact are represented in Table 2. 

Table 2 

Leadership Practices for Equity Identified by Ross & Berger (2009) 


Curriculum 

Interpretation 

Encourage staff members to talk about issues of diversity and social 
justice 

Model equity beliefs for staff 

Clarify misconceptions about equity 

Create a safe, affirming school environment 

Instructional 

Practices 

Enable teachers to provide students with the support they need 

Provide all students with access to the whole curriculum 

Recognize the potential for bias in special education identification 

Support research-based instructional strategies 

Assessment 

and 

Evaluation 

Monitor progress toward achievement gap reduction 

Use appropriate accommodations for assessments 

Discourage strategies that involve gaming the accountability system 
Celebrate all achievement gains 

Increase the reliability of assessments for diverse student populations 
Avoid cultural, linguistic, and gender bias in tests 

Community 

Involvement 

Recognize the expertise of parents and community members 

Create partnerships with parents to support learning 


METHODOLOGY 

As a utilization-focused program evaluation design (Patton, 2002), the study began with the 
development of an evaluation framework in partnership between the PRN a research team at 
the university. Ligure 1 depicts the framework in the form of a Theory of Action (TOA). The 
goal was to use the program evaluation process as an opportunity to develop a consistent cycle 
of inquiry in which the preparation program staff could collect, analyze, and use data for the 
purposes of improving the program and modeling the inquiry process taught to aspiring 
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principals to use in schools (Love, 2009). Therefore, the TO A/Evaluation Framework 
articulates the Enabling, Intermediate, and Long-Term program outcomes; the indicator data 
collected on the outcomes (bulleted in italics); and the timeline for data collection. Phase I of 
the study consisted of administering a survey questionnaire to recent graduates of the program 
and collecting assessment data and feedback from current PRN participants accessible at the 
time of the program evaluation (see indicator data underlined in Figure 1). Phase II was 
designed to further explore recent graduates’ perceptions of the nature and relevancy of the 
program in developing their commitment and skill to lead for equity, an intermediate outcome 
in the TOA/Evaluation Framework. This section will describe the methodology for Phases I 
and II. 


Intermediate Outcomes Long-Term Outcomes 


PRN Practices 


Structural 

• partnerships 

• supportive district/state 
infrastructure 

• monitoring for 
improvement 

• support for participants 

• rigorous entrance 
requirements 

Content/Curriculum 

coherent, standards-based, 
individualized curriculum 
focused on instructional 
leadership for equity 

Pedagogy/delivery 

• intensive residency 

• cohort structure 

• mentor support 

• authentic assessments 

• problem-based learning 

• reflection 

• modeling adult learning 


Enabling Outcomes 

Measure annually 

Caliber of Participants 

•Ratings on aspiring and 
mentor principal 
admission rubrics 

Performance on 
Program Experiences 
and Assessments 
• All rubric ratings 

Proficiency and Growth 
in Leadership 
Standards 

•Final exhibition scores 
•Pre and Fined mentor 
assessment 

Quality of Program 
Experiences 

• Mentor feedback 
• Grad Survey 


Measure 2-3 years 

Commitment/Skill to 
Lead for Equity 

• Grad focus group 

Completion of Program 

• Graduation rate 

Obtaining Leadership 
Positions 

• Program database and 
Grad Survey 

Participation in Post 
Graduate Professional 
Development 

•Grad Survey and 
program databases 

Proficiency in National 

Leadership 

Competencies 

•Score on ETS exit exam 


Measure every 5 Years 

Increased Student 
Achievement in Schools 
Lead by Graduates 

• Growth rates compared 
to similar schools 

Reduced Equity Gaps in 
Schools Lead by 
Graduates 

•Reduced gaps between 
subpopulations 

Improved School 
Learning Environment 
in Schools Lead by 
Graduates 

•State survey data 

Increased Quality of 
Educational Leaders in 
Statewide Community 
of Practice 

•State educator 
evaluation data or 
preparation program 
report card 


Figure 1 . Theory of Action/Evaluation Framework 

Note. The indicators of the outcomes are in italics. As part of this study, data was collected for the underlined 
indicators. Data was not available and/or will be collected at a later date for the other indicators. 


Phase I: Data from Program Databases and Assignments 

Prior to the period of this program evaluation, the PRN program only tracked completion of 
participants in the program, current roles of graduates of the program, and data from graduate 
surveys. Theses data were not adequate for program staff to engage in an ongoing cycle of 
inquiry to improve the program. Therefore, to measure the Enabling Outcome, Performance 
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on Experiences and Assessments, detailed rubrics were created for each learning experience 
and assessment, and program staff calibrated their scoring on all assessments throughout the 
study year to insure inter-rater reliability was high. Likewise, to measure another Enabling 
Outcome, Proficiency and Growth in the Leadership Standards, the rubrics used during 
participants’ final exhibition and by the mentor principals’ to rate aspiring principal 
participants’ proficiency were revised, articulated across performance levels, and program 
staff and mentor principals engaged in calibration exercises. Finally, the Enabling Outcome, 
Quality of Program Experiences, was measured in the survey discussed in the subsequent 
section and by analyzing the mentor principals’ written responses on quality, challenges, and 
improvements for program. The data collected for these four TO A/Evaluation Framework 
Outcomes were compiled and used by program staff to make program adjustments at the mid¬ 
point and at the end of the year. 

Phase I: Data from Graduate Survey 

Two previous graduate surveys had been conducted prior to this study, the first in 2005 
(iV=21) and the second in 2009 (N= 6); therefore, the same survey was sent out to all 
participants who had graduated in the past three years (N= 21). Of those graduates mailed 
questionnaires, 66% (/V=14) completed and returned them. The PRN Graduate Survey was 
designed to collect data on graduates’ perceptions of quality, challenges, and improvements 
for the program. Four of the Intermediate Outcomes, Quality of Program Experience, 
Obtaining Leadership Positions, Commitment/Skill to Lead for Equity, and Participate in 
Ongoing Professional Development were measured by data collected from survey items. To 
determine content validity, the survey underwent content review by four educational 
leadership professors and instructors. 

Descriptive statistics were run for all the items that contained a rating scale; for all the 
open-ended items, the written responses were compiled. The means for the items that 
measured the Enabling Outcome, Quality of Program Experience, were compiled in a 
spreadsheet that contained means for the same items from the 2005 and 2009 surveys. The 
means for all three years by item were represented together to allow for trend analysis. 

Phase I: Data from State Database 

To measure the Long-term Outcome, Increased Student Achievement in Schools Lead by 
Graduates, data were gathered from a state database. The Rhode Island Department of 
Education databases (RIDE, 2012) were used to collect data on student achievement in 
schools lead by program graduates. PRN graduates who had been a principal or instructional 
leader (e.g., Director of Curriculum, Co-Principal) in the elementary or middle school during 
at least the three-year period between 2008-2011 ( N= 20) were included in the study. Student 
achievement data were represented by the mean of the index proficiency scores for all 
students in a school on the English Language Arts (ELA) and Math New England Common 
Assessment Program 2008/09 and 20010/11 exams (RIDE). The school level (elementary or 
middle) and type of school (suburban, urban ring, or urban) were entered in a database with 
the index proficiency scores for all program graduates who had been a principal in the three- 
year period and for every school in the state to use as comparison groups (N= 225). After the 
data were disaggregated by level and type of school, to measure the growth or increase in 
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student achievement, the means from the 2008 and 2011 index proficiency scores were 
compared for the PRN (N= 20) and for everyone other school in the state. Due to the small 
number of program graduates once disaggregated, inferential statistical analysis is not 
possible; however, since this is a critical Long-term Outcome, it was important for the study 
to note as descriptive data. 

Phase II: Data from Focus Group 

Phase II participants included program graduates from 2008-2011 who were sent the PRN 
Graduate Survey and agreed to attend the focus group session. The final group ( N=7 ) 
represented a cross-section of class years, school types, and professional backgrounds. A 
focus group moderator’s guide was developed for the focus group session, using standard 
formatting for the sequence of questions (Krueger & Casey, 2009). Eight questions for this 
topic were designed to range from demographic and employment information to personal 
reflections and assessments of how the program prepared participants to ‘lead for equity’. 
Specific content questions regarding barriers or enhancing experiences that assisted 
participants with their professional development, and questions regarding their mentor 
relationships, papers and exhibitions, and cohort/network meetings further added to the 
discussion. The final question was designed to solicit ‘advice’ from program participants 
regarding program strengths and weaknesses and encourage debriefing as a way to conclude 
the conversation without emotional conflict or discomfort. 

The data collection procedures comprised of a single focus group session, conducted 
in late spring on University premises, facilitated by an external moderator for a 90-minute 
session. The session was audio taped, consent forms were distributed and ‘ground rules’ were 
conveyed prior to the beginning of the questioning. 

Focus group data were transcribed following the session, and raw data files were 
analyzed using a sequence of coding, content analysis and thematic clustering. Modifying 
Krueger and Casey’s (2009) Classic Approach for focus group data analysis, and Miles and 
Huberman’s (1994) three-tier coding strategy (descriptive, interpretative, and pattern codes), 
data analysis process encompassed ‘chunking’ the data until it could be transformed into 
themes and categories, using participants’ words and expressions to illustrate their ‘meaning 
essence’ (Miles & Huberman, 1994). The final stage of analysis involved reviewing 
transcripts to connect participant’s quotes, stories and expressions with the thematic clusters 
in order to develop a narrative that reflected the findings (Krueger & Casey, 2009, p. 122). 
Trustworthiness strategies included ‘’thick descriptions’, peer debriefing, and member¬ 
checking to ensure rigor and credibility in the results (Lincoln & Guba, 1985). 

KEY FINDINGS 

Phase I Findings from Program Assessments, Feedback and Graduate Surveys 

Enabling Outcome: Performance on Program Experiences and Assessments. Data 
were compiled from rubric rating scales for each major assignment completed by aspiring 
principal participants in the program during the 2011/12 school year (iV=18) and presented by 
ranked mean in Table 3. Aspiring principals must earn a “3” on each assignment to complete 
the program. If they earn less, on many assignments they have a limited opportunity to revise 


166 



and resubmit the work. One notable finding is that the top five experiences and assessments in 
Table 3 on which participants scored highest were either conducted or turned in toward the 
end of the year. Though not apparent in the data in Table 3, another interesting result was that 
the Learning Plan and the Action Research Paper had the most amount of revisions required 
for participants to earn at least a 3. Both assessments require authentic leadership and ongoing 
effort to engage in the complicated assignment structure for aspiring principals to demonstrate 
learning. Due to the low initial scores on these two assignments, major revisions were made to 
the instruction to increase student success. The intention is to continue to compile and analyze 
the data each year to look for trends and make improvements. 

Table 3 

Performance on Program Experiences and Assessments (N= 18) 


Program Experiences and Assessments 

Mean 

School visits 

4.0 

End-of-Year Exhibition 

4.0 

Narrative Reflections 

3.8 

Clinical Visit 3 

3.7 

End-of Year Paper 

3.6 

Summer Residency 

3.5 

Fall Shadowing 

3.5 

Learning Plan 

3.5 

End-of-year Mentor Assessment 

3.5 

Readings 

3.4 

Mid-year assessments 

3.4 

Action Research Paper 

3.3 

Networking/Formal Learning 

3.2 

Portfolio/evidence 

3.2 

Vision Paper 

3.1 

Mentor meetings/coaching 

3.1 


Note. Scale associated with all items was 1 inadequate, 2=approaching, 3=adequate, and 
4 =distinguished. 

Enabling Outcome: Proficiency and Growth in Leadership Standards. This outcome 
was measured with two data sources. The first was from the final exhibition scores for 
aspiring principals completing the program in 2012 (V=14). Final exhibitions are a time for 
participants to articulate what they have learned and how they learned it around each 
leadership standard. The exhibitions are scored on a 4-point scale by every PRN staff 
member, mentor, and aspiring principal present for the exhibitions. The scores are averaged 
by standard (see Table 4). All six standards are well above the adequate level, with Standards 
1 and 2 being the highest. Standards 1 and 2 contain practices that are the heart of what is 
entailed in instructional leadership. Most of the participants in the PRN come with a large 
degree of instructional leadership experience, as this is a pre-requisite looked at closely in the 
admittance screening process. Standard 6 was the lowest score in Table 4. The Educational 
Systems Standard includes practices that require leaders to engage in dialogue and advocacy 
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outside of their schools. These practices are more challenging for aspiring principals to 
engage in during their residency year(s). 

Table 4 

Final Exhibition Scores by Standard ( N=14) 


Leadership Standards 

Mean 

Standard 1: Mission, Vision, Goals 

3.7 

Standard 2: Learning and Teaching 

3.7 

Standard 3: Managing Systems 

3.6 

Standard 4: Collaborating 

3.6 

Standard 5: Ethics Integrity 

3.6 

Standard 6: Educational Systems 

3.3 


Note. Scale associated with all items was l=inadequate, 2=approaching, 3=adequate, and 
4= distinguished. 

The second piece of data collected to measure the outcome. Proficiency and Growth in 
Leadership Standards, were the pre and final mentor assessments of the aspiring principal 
participants completing the program in 2012 (V=14) using a 4-point scale on the leadership 
standards rubric. Each aspiring principal has a mentor principal with whom they work closely. 
The PRN considers the mentor the primary instructor for the aspiring principals. Therefore, 
the mentor is in the best position to rate the proficiency of the aspiring principals throughout 
the year. While there are other data to triangulate aspiring principals’ proficiency (e.g., 
exhibitions, portfolios, papers), the mentor rating provides critical feedback and perspective. 
Mentors rate aspiring principals on all the sub-sections of the standards at the beginning, 
middle and end of the year. Each standard’s sub-section were averaged and the means for the 
pre-assessment, the final assessment, and the growth (difference between pre and final) are 
represented in Table 5. 

All standards were rated at or near approaching at the pre-assessment and all standards 
were adequate at the final assessment. The mentor’s pre-assessments validate the PRN 
screening process because the highest two standards are 1 and 2, which are the core practices 
for instructional leaders. The lowest scoring pre-assessments are for Collaborating with 
Stakeholders (Standard 4) and Educational Systems (Standard 6), though these two standards 
had some of the highest growth. Both these standards are difficult to enact as a classroom 
teacher, and the residency as an aspiring school leader gives great opportunity to engage and 
learn in the broader educational arenas represented by these standards. Interesting, the highest 
scoring final assessment is for Standard 5 (Ethics and Integrity). As scholars like Sergiovanni 
(1992) have advocated, the heart of school reform work for a leader has a heavy moral 
component. In the PRN, a major focus of the preparation is on enabling participants to 
become equity-oriented leaders (Skrla, McKenzie, & Scheurich, 2009) which requires a great 
deal of personal growth (articulated further in focus group findings), as well as leadership that 
models and challenges others to move schools from being agents of social reproduction to 
forces for social change (Ross & Berger, 2009). 
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Table 5 

Mentor Ratings of Aspiring Principal Participants by Standard (iV=14) 


Leadership Standards 

Pre-Assess 

Mean 

Final Assess 
Mean 

Growth 

Mean 

Standard 1: Mission, Vision, Goals 

2.4 

3.7 

1.3 

Standard 2: Learning and Teaching 

2.2 

3.5 

1.3 

Standard 3: Managing Systems 

2.0 

3.5 

1.5 

Standard 4: Collaborating with Stakeholders 

1.9 

3.6 

1.7 

Standard 5: Ethics Integrity 

2.1 

3.8 

1.7 

Standard 6: Educational Systems 

1.7 

3.3 

1.5 


Note. Scale associated with all items was 1 inadequate, 2=approaching, 3=adecjuate, and 
4 =distinguished. 


Enabling Outcome: Quality of Program Experiences. This outcome was measured 
through two data sources, the first of which was from written feedback solicited from current 
mentor principals in the program (V=18). The feedback was organized into four categories 
represented in Table 6. The mentors recognized many of the core practices of the program 
(see Table 1) as valuable, such as the cohort structure, the authentic residency, the practice of 
reflection, and the use of standards with individualization which can be seen in the use of a 
learning plan and coaching from an advisor. The universal concern mentors have is time to 
give their mentee, as well as the program. The areas of self growth mentors identified show 
that the nature of the learning relationship between mentors and aspiring principals is 
reciprocal. As with the rest of the outcomes data presented, the data on the improvments the 
mentors recommended have already been used to make changes to the program. 

Table 6 

Feedback from Mentor Principals in 2012 


Mentors Identify as Valuable about the Program 

• Cohort structure and meetings and networking with colleagues of diverse perspectives 

• Residency is authentic leadership practice for aspiring principals 

• Systematic reflection throughout the program, including mid-year work 

• Learning Plan structure to guide the learning, and alignment with standards 

• Mentor Standards used as a guide 

• Resources of PRN Advisors as coaches 

• Bigger impact of PRN on school as a resource to build leadership/leaming communities 
Mentors Identify as Challenges to the Program 

• Taking time to participate while balancing growing priorities 
Mentors Identify as Improvements Needed to the Program 

• More Sharing with Mentors on Aspiring Principals (AP) progress and learning 

• More/continued use of protocols to get feedback from their PRN cohort 

• Create more cohesion within cohort since it feels larger, less ‘homey’ 

• Have ways to make up missed network meetings 
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Mentors Identify as Areas of Growth for Themselves 

• Skills in distributing leadership and building learning community 

• Prioritizing time for reflection 

• Own overall leadership practice - Learned alongside AP 

• Evaluating AP learning 

• Scaffolding/coaching and showcasing APs learning and work 


The second data source to measure the Quality of Program Experiences was from the 
graduate perspective and collected through the PRN Graduate Survey distributed to 
participants who had graduated between 2008-2011 (A=14). The same survey had been 
administered to previous graduates of the program in 2005 and 2009. The results displayed in 
Table 7 are from the items that asked participants to rate on a 5-point scale the degree to 
which each of the program experiences and assessments gave them the knowledge and skill to 
be prepared to lead change in schools. Table 6 contains the mean responses for all program 
experiences for all three administrations of the survey and ranked by the mean of all three 
surveys for each experience. 

The vast majority of the program experiences were rated at a considerable extent or 
great extent throughout all three administrations of the survey. Notably, a few of the 
experiences were rated highest across nearly all three years: Internship at school. Learning 
relationship with mentor, and PRN Advisor visits and feedback. All three of these speak to the 
importance of the authentic learning that happens through the residency with supportive 
mentoring and coaching that happens from the mentor and advisor. The lowest-rated 
experience in the 2005 and 2009 PRN Graduate Surveys was the Feedback Circle; however, 
after improvements were made based on the survey data, this experience was improved. The 
Feedback Circle was originally a small group of colleagues that the aspiring principal was 
supposed to convene to ask for feedback on their leadership; however, the structure was 
unclear. The Feedback Circle experience was modified so that aspiring principals identify the 
core group of colleagues that they will be working with to implement the initiative in their 
Action Research. Aspiring principals have to continuously find ways to get buy-in, grow 
commitment, change practice, and get feedback from this group. This reinforces the practices 
of shared instructional leadership (Marks and Printy, 2003) and makes the concept of a 
feedback circle more relevant. 

Table 7 

Graduate Ratings on Degree Experiences Prepared Them to Lead Change 


PRN Experiences/Assessments 

2005 Survey 
M(N= 21) 

2009 Survey 
M (N= 6) 

2012 Survey 
M(N= 14) 

All Surveys 
M 

Internship at school 

5.0 

5.0 

4.8 

4.9 

Learning relationship with mentor 

4.4 

4.8 

4.6 

4.6 

PRN Advisor visits and feedback 

4.7 

4.6 

4.6 

4.6 

Required readings 

4.5 

4.4 

4.5 

4.5 

Final Exhibition and feedback 

4.3 

4.4 

4.5 

4.4 

Mid-year Exhibition and feedback 

4.1 

4.4 

4.7 

4.4 

Action Research Project 

4.4 

4.4 

4.4 

4.4 

Meetings with AP’s & Mentors 

4.5 

4.2 

4.5 

4.4 
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Vision Paper 

4.5 

4.4 

4.3 

4.4 

Portfolio, review, and feedback 

4.7 

4.4 

4.0 

4.4 

Non-PRN workshops/trainings 

4.4 

4.6 

4.1 

4.4 

Final Paper/feedback 

4.4 

4.2 

4.4 

4.3 

Learning Plan 

4.6 

4.2 

4.1 

4.3 

Reflections 

4.3 

4.6 

3.9 

4.3 

Meetings with AP’s only 

4.5 

3.6 

4.7 

4.3 

Mid-year Paper/Assessment 

4.1 

4.2 

4.3 

4.2 

Visits to other schools 

4.1 

4.2 

4.0 

4.1 

Mid-year Mentor Paper/Assessment 

4.0 

4.4 

3.9 

4.1 

Final Mentor Ratings 

4.0 

4.0 

3.8 

3.9 

Feedback Circle 

3.5 

3.6 

4.1 

3.7 

Note. Scale for all items: 1 =Not at all, 2 =a 

little, 3= 

-somewhat, 4=a 

considerable extent, 5 

=a 


great extent. 

Intermediate Outcomes: Completion of the Program and Obtaining Leadership 
Positions. Since inception, the program has monitored completion data and the positions that 
graduates assume after completing the program. From 2000-2012, the program has had a 99% 
completion rate. Of the 83 graduates, 90% (N=15) have been hired into building, district or 
non-profit leadership roles. The remaining 10% (V=8) have remained in teacher leadership 
roles. The types of current leadership roles the graduates (V=83) have assumed are displayed 
in Table 8. The majority (77%) of graduates are currently in school or district administration 
or coaching positions in Rhode Island. Most of the remaining graduates, with the exception of 
those who have remained teacher leaders or retired, have moved from building leader 
positions to other leadership roles at the district level or in non-profit organizations and higher 
education. 

Table 8 

Current Leadership Roles of Graduates 


Leadership Roles 

N 

% 

Principal/Director 

31 

37% 

Assistant Principal 

16 

19% 

District Administration 

9 

11% 

Instructional Coach/Coordinator 

8 

10% 

Teacher Leader 

8 

10% 

Retired/Moved Out-of-State 

6 

7% 

Director/Administrator at Educational Non-Profit Organization 

3 

4% 

Higher Education Administrator/Instructor 

2 

2% 


Long-Term Outcome: Increased Student Achievement in Schools Lead by 
Graduates. State assessment data was used to compare student achievement between PRN 
graduates’ schools and non-PRN graduates’ schools. Table 9 and Table 10 represent the 
student achievement data for PRN graduates’ who had been a leader in their school between 
2008-2011 (V=20) compared to similar demographic schools. Due to the small sample size, 
the findings for this outcome are descriptive, as no statistical significance tests were 


171 






appropriate to conduct. In five out of eight areas, program graduate schools evidenced greater 
growth than comparison schools: Urban Ring Elementary Schools in ELA, Urban Middle 
Schools in ELA, Suburban Elementary in Math, Suburban Middle in Math, and Urban Middle 
in Math. 


Table 9 

Comparison ofPRN Graduates to Non-PRN Graduates ’ School English Language Arts (ELA) 
Scores on a State Assessment in 2008 and 201L 


School Category 

N 

ELA 2008 
M 

ELA 2011 ELA Growth 

M between 2008-2011 

PRN Suburban Elementary 

9 

92.16 

93.36 

1.20 

Non PRN Suburban Elementary 

62 

91.44 

93.35 

1.90 

PRN Urban Ring Elementary 

6 

89.48 

92.55 

3.06 

Non PRN Urban Ring Elementary 

49 

89.07 

90.96 

1.88 

PRN Suburban Middle 

3 

92.55 

94.71 

2.16 

Non PRN Suburban Middle 

24 

91.45 

94.02 

2.57 

PRN Urban Middle 

3 

75.65 

83.63 

7.98 

Non PRN Urban Middle 

11 

77.79 

83.72 

5.93 

Table 10 





Comparison ofPRN Graduates to Non-PRN Graduates ’ School Math Scores on a State 

Assessment in 2008 and 2011. 





School Category 

N 

Math 2008 
M 

Math 2011 
M 

Math Growth 
between 2008-2011 

PRN Suburban Elementary 

9 

90.71 

92.99 

2.28 

Non PRN Suburban Elementary 

62 

89.31 

91.45 

2.14 

PRN Urban Ring Elementary 

6 

84.76 

86.40 

1.64 

Non PRN Urban Ring Elementary 

49 

84.02 

85.70 

1.68 

PRN Suburban Middle 

3 

88.22 

90.93 

2.71 

Non PRN Suburban Middle 

24 

87.51 

89.44 

1.93 

PRN Urban Middle 

3 

63.63 

72.97 

9.33 

Non PRN Urban Middle 

11 

70.30 

72.70 

2.40 


Phase II Findings from Focus Group 

Phase II findings are reported according to the inter-related elements known to affect 
participants’ engagement and satisfaction with the program. Results are presented in the 
participants’ own words, capitalizing on the stories, details, and multiple realities that were 


172 





expressed in interactive discussions during the focus group session. The five key themes that 
emerged from the findings are presented. 

Theme #1 Making Hard Decisions and Bringing People Along!: Residency 
Experiences, Challenges and Rewards. Participants shared detailed accounts of their 
residency placements. Most participants described the situations that arose as instances where 
they could see the direct relationship between what they learned in the program and their 
ability to navigate difficult situations ‘on the job’: 

■ “You start the program with certain knowledge and readings and a certain picture in 
your head about what you are going to do in your residency, and then things happen 
that change that picture... and you look back and say, ‘Gee! I thought I was going to 
do one thing and I ended up doing another and its really ok!”’ 

■ “You take what you leam in the program and you try to help people become aware of 
their biases - how they are not being equitable, and you start pushing those things as a 
leader ... and you start to see that people were afraid to own those biases... but you 
eventually see the fruits of your efforts...” 

Participants also described the challenges of their residency experiences. Most 
graduates found that finding the balance between two jobs, plus the program, plus their 
personal lives, was an enormous obstacle to overcome; as one participant expressed, “I think 
that one of the things that was just so hard was balancing two jobs... and not letting either one 
suffer...”. Equally challenging was ‘finding one’s place’, or trying to envision oneself in a 
leadership role different from prior positions or experience. One PRN graduate described the 
duality as “What hat are you wearing? What alliances are you honoring?”, followed by her 
later conclusion that “...now that I look back at it, it did prepare me for the eventual break 
from the past into a new leadership role.” 

A common sentiment was learning how to make the difficult decisions, especially 
when one’s role had shifted within a community in which they had long resided: 

■ “At the end of the day, when hard decisions needed to be made and hard conversations 
needed to happen, you look towards the goal: you look towards the mission of your 
school and your vision and that is compelling...” 

■ “I came into the leadership role from the perspective as a classroom teacher and it 
wasn’t quite as easy as I thought. You see things from multiple perspectives all of a 
sudden, and then you realize what leadership is like, what it’s like to be in the 
classroom, and you marry the two together... to be an effective leader of children, of 
teachers, and of colleagues.” 

Theme #2 Walking a Fine Line - Finding the Balance: Mentor Relationships. As 

one participant stated, “The mentor relationship is complicated!”. Communication between 
and among participants and their mentors played a vital role in the development of each 
individual as a professional, a leader, and a contributing member of their educational settings. 
Conversely, participants described the numerous challenges that accompanied the substantial 
benefits of working with their mentors. As another participant stated, “these mentors are 
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grooming you to be their next assistant, so they have a vested interest in your success... they 
don’t want you to look foolish, so their investment involves being careful about what you do 
or don’t do while you are there...”. 

While working with mentors created important professional opportunities for each 
participant, these relationships also generated tensions over power and control, embedded in 
the experiential learning that occurred. Participants acknowledged the need to learn by doing, 
and they admitted that the opportunity to assume a leadership role was sometimes difficult, 
depending on the mentor. As one PRN graduate expressed, “you would start to facilitate a 
discussion with a parent, and then you would be stepped on, and suddenly you were no longer 
facilitating the discussion...”. A range of sentiments about these control issues included the 
following sentiments: 

■ “I remember one of the questions during our interview, when they were asking us 
about the partnership... and my mentor was asked, directly, whether they could ‘let go 
of control?’ and my mentor honestly expressed his concern by saying ‘it’s going to be 
a challenge!”’ 

■ “We had a lot of difficult conversations about the experiences I needed to have, and 
there was talk... ‘oh, yes! I will let you have more control’, but it never came to 
pass...” 

■ “I saw how my mentor struggled with control and I realized that if I had someone 
under me, working in this way, I am not sure I would be able to let go and allow 
someone else to run that conference or talk to that parent...” 

This struggle brought with it a greater sense of self-awareness, for both parties: 

■ “The flip side of this experience is that the mentors have the opportunity to learn just 
as much as the aspiring principals... if they let themselves learn. If they are willing to 
open up and let learning take place.. .a kind of double-edge sword that would be 
difficult for any of us” 

The conversation about mentor relationships, both positive and negative, generated 
discussion about the options for improving that piece of the PRN experience. Several 
participants advocated for ‘multiple mentors’ or a different screening process for mentors; as 
one participant noted, “...there is a lot to be said for having multiple mentors, multiple 
viewpoints on how to manage situations and assume leadership”: 

■ “It is complicated and I know this is an intense program but the idea that you are 
learning from just one person, and that somehow you are locked into one set of 
ideas... well, that may not be the best option?” 

■ “Why couldn’t mentorship come from different sources, by having several mentors?” 
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■ “I think having multiple mentors was what ended up happening to me naturally, just 
because of my situation... and I learned a lot, sometimes different things, from both of 
them...” 

Participants transitioned from this concept of having several mentors to the focus on how to 
conduct screening of potential mentors, and offered the following recommendations: 

■ “I would recommend that just as the aspiring principal needs to go through the 
screening process, so should the mentors... they are principals in schools and are 
under a lot of pressure.. .There should be a sense of what the relationship absolutely 
must involve and what is really needed for true mentorship...” 

■ ‘‘Maybe someone in a different school or a different role can balance what you are 
already doing with your mentor to add to what you are learning... and the principals, 
as mentors, have so many demands on their time that to expect them to provide all the 
knowledge and experience you need may be unrealistic?” 

Theme #3 Focused Hard Work and Reflection!: PRN Program Components. The 

majority of participants indicated that while they spent considerable time either in class or in 
preparing for class, the value of the total experience could not be over-stated. The intentional 
program design, which incorporated projects, papers, reflection, and presentations, comprised 
just a few of the important elements of this experience: 

■ “One of the special things about the program is that it is individualized, based on who 
you are and where you are and the needs of the school and district you represent.. .it is 
one of the greatest benefits of the program!” 

■ “The specifics of the program, the papers, the tools, the learning devices, the 
portfolios... we could extract data from the schools and apply what we had learned... 
it was immediately transferable!” 

■ “In a way, it is still an obstacle for me... trying to move something or use what I 
learned in PRN, to increase equitable outcomes... having people realize their own 
influence in that perspective and realizing that people are afraid to own ‘equity’ and 
that their role is to recognize that they can influence and lead that effort...” 

These sentiments mirror most of the comments offered during the discussion, to 
include a significant element focused on personal reflection: 

■ One of the biggest pieces of the program that I took away with me was the need for 
reflection.... Reflection, reflection, reflection! It has taught me to take the time to stop 
and think, so that when something happens, I need to process and not just default, go 
to the typical reaction...” 

■ “Reflection is a huge piece of this program. I think as educators we often go, go, go - 
and we don’t stop to take the time to think about what was happening. Leadership 
requires that reflection and the program taught you that...” 
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Another significant piece of the PRN program was the immersion in the leadership 
standards, which participants acknowledged as an important element in their learning process: 

■ . .getting really grounded in the leadership standards and the true meaning behind 
those standards, impacted everything you did in school...” 

■ “Focusing on the standards in this program really grounded me, really made me pay 
attention to what was critical and important in my role as a leader” 

Participants discussed the value of the vision paper assignment, which they prepared at 
the beginning of the PRN process: 

■ “Early in the program you write down your vision in a paper, and when you look at the 
guidelines, and you try to outline your vision you say ‘huh?, what?’, but then you get it 
down on paper and you look at your final product and you say ... Wow!” 

■ “My vision paper was different, I think I came with a set of assumptions... this is what 
I felt education was about. But I was looking through the eyes of a parent, and after I 
had gone through the PRN, had the experience, done the research, had the discussions, 

I looked at what I wrote and... oh, my gosh! My vision was so different than what I 
initially thought and I realized that I had grown up!” 

Another important component was the experience of the cohort as a learning 
community: 

■ “I remember thinking at the time, when we were going through the exhibitions and 
getting our portfolios ready, that there was wonderful value in seeing this all 
together... and I was dying for the opportunity to have time one day when we would 
all meet and just sit around a table and talk about the material we shared and learned 
from...” 

■ “The professional learning communities that formed in our peer group and then going 
back to school during that year, trying to apply what we learned... it was significant” 

The overarching sentiments, however, were expressed in how hard the work was and 
how gratifying the program experience was for each individual: 

■ “I came into this program saying to myself, ‘how hard can it be?’, but .. .its complete 
immersion! Focused hard work!” 

■ “For me, it was the structure of the program, the projects, the way we would read 
something and reflect on it, and have a concentrated amount of time to apply those 
concepts... and it was through the application that you could see the big picture. The 
leaming-by-doing had the biggest impact on me and that came from the structure of the 
program.” 
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■ “What happens to you during the process is that you look back and think about the various 
barriers and you realize that the program helped you get through it.. 

Participants indicated a high level of satisfaction with the design of the program, 
directed towards building capacity for ‘leading for equity’; their perceptions of the value of 
this program was viewed as integral to their satisfaction with their experience, overall. As one 
participant summed it up, “I did not know what to expect, and I was so quickly moved into the 
program... but once I got over the ‘shock’, I felt so empowered when I came back out!” 

Theme #4 A Special Kinship and Built-in Empathy: Cohort Interactions and 
Experiences. Participants were not haphazard in their allotment of time; they quickly learned 
that time management was an essential ingredient to their success, both academically and 
professionally. Part of that management was linked with their cohort relationships, the ways in 
which they shared the experience and worked together. Many found that their peer-to-peer 
interactions formed the basis for their satisfaction with the experience, finding empathy and 
camaraderie in each other; the findings here suggest that these individuals intuitively set aside 
time for interactions with each other. As mentioned earlier, participants identified the cohort 
experience as connected to the development of a professional learning community, a special 
network of colleagues: 

■ “We had a very close cohort, you gain such a kinship ... it’s like going through the 
birth of your children and its intensive... you laugh, you cry, together you support one 
another and it is all encompassing.... And then it is done! You intend to stay 
connected, but it is so hard... I miss them all!” 

■ “We all came from such different places, and I crave that comradeship and I went to 
the Equity Institute last year just to connect with people again and feel that same 
feeling...” 

■ “Having the close cohort connection was l ik e built-in empathy...” 

As one participant indicated, while others agreed, the transition of classmates to 
colleagues to friends during the program was a meaningful benefit. As she noted, “...I might 
want somebody to talk to as I move ahead, and we all had each other to share things with... I 
remember [the program director] coaching us to do that, once you leave here, seek each other 
or a group that is going to help you down the road, problem solving and seeking advice...” 

Theme #5 Strengthening a Strong Program : Recommendations for Program 
Modifications and Improvements. Participants were quick to praise the program and their 
experiences; suggestions and recommendations were provided in the context of how to 
strengthen an already strong and vibrant program. As one participant stated, “...there are so 
many PRN graduates, there’s a voice, a strong voice, and collectively we should be able to 
support the program going forward and provide a network for [the program director] and the 
program development...”. 
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Specific program recommendations included the following ideas: 

■ Screening process for mentors 

■ Multiple mentors to broaden participants’ experience 

■ PRN grads as mentors for PRN students 

■ Support group after graduation 

■ Alumni network expanded 

■ Ongoing professional development 

■ Networking opportunities for graduates 

Finally, there was consensus around the sentiment that the PRN had been more than an 
educational experience for participants; it had been transformational. Participants expressed a 
range of emotions regarding their overall PRN experience: 

■ “When you are immersed in the program we would cry together or complain and ask 
why we had to do something, and then... you look back and you make those 
benchmarks and achieve those accomplishments and you say Oh!! Now I get it, now I 
know why we did that ... Trust the program! It’s rigorous but so valuable...” 

■ “The process, for me, was truly transformational. I think it is just so moving when you 
are going through something where you have to stand up in front of your colleagues 
and state what you believe... and I started to cry and could not stop crying... actually, 
it was part of the transformation, the growing, being really honest about what you 
believe. It brought it all together for me!” 

■ “It was a wonderful experience, really life transforming! I think even career 
transforming, for me, and yeah... the integrity of the program is solid but there are 
always things that can be improved. But I am very, very proud to be a graduate of the 
PRN and I look to [the program director] and the way she structured it and the way she 
ran it, with all her pressures, and she did an exceptional job!” 

CONCLUSIONS AND RECOMMENDATIONS 


A Strong Program 

The results from Phase I of the study demonstrate that the program is achieving strong results 
in regards to participant performance, growth, completion, and attainment of post-graduate 
leadership roles. Further, the survey results indicate that participants feel that nearly all 
program experiences had a considerable or great impact on their ability to lead change, and 
this has been relatively consistent over the 12 years of implementation. Phase II results from 
the focus group concur with the Phase I results that suggest the program is high quality. While 
focus group participants offered suggestions for improvement of the program, many of the 
comments and themes reflect that the overall perception was that the program was highly 
effective in preparing them for their leadership roles. 
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Interconnected and Coherent Equity-Focused Program 

The focus group findings revealed that the structure of the program, combined with the 
residency experience, provide a comprehensive and transformational experience for 
participants. As one participant said, “It was a wonderful experience, really life 
transforming...even career transforming.” The degree to which graduates rated the PRN 
experiences all fairly high on the PRN Graduate Surx’ey also indicates that they feel the 
structure, curriculum, and pedagogy of the program are interconnected and provide a coherent 
learning process. Results suggest that participants recognized that a major focus of the 
program is on preparing equity-oriented leaders. Further, focus group participants did not see 
this focus as isolated or discrete, rather it was deeply integrated into all their learning 
experiences. 

Mentoring 

A cornerstone of the program is the intensive residency experience with a mentor. This can be 
seen in how high both of these experiences are rated in the PRN Graduate Survey results. The 
intensity of the experience and the relationship lead to powerful learning, and can lead to 
challenges. The written mentor feedback identified enriching experiences, like the 
professional development with the cohort and the learning plan, and the challenges they faced, 
mostly around time and a sense of community with the group. Focus group participants 
offered specific and practical suggestions for program modifications to improve the mentoring 
component. Additional study by the PRN in these areas is strongly suggested, particularly 
regarding the idea of using multiple mentors and using PRN graduates as possible mentors. 

Cycle of Inquiry and Modeling 

The Theory of Action/Evaluation Framework designed at the onset of the study was used to 
develop an ongoing cycle of inquiry to improve the PRN program. PRN staff used the data 
collected to measure the Enabling Outcomes at the mid-point of the year to make mid-course 
corrections and at the end of the year to make improvements for the 2012/13 school year. The 
PRN staff also compiled the data from this evaluation into a data dashboard that was 
presented to the Center for Leadership and Educational Equity (CLEE) Board of Directors as 
a tool to track progress of the PRN toward achieving the outcomes described in the 
TOA/Evaluation Framework. This process will happen each year as new data is available so 
that trends can be analyzed and so that the CLEE Board can use the data to inform decisions 
around providing resources and seeking funding. Continuing the cycle of inquiry using the 
TOA/Evaluation Framework developed in this study will be essential in order to monitor 
important outcomes and to improve the PRN. 

There is evidence from the mentors’ written feedback and the graduates’ focus group 
results that participants have found multiple ways to use learning experiences modeled by 
PRN staff back in their school communities. The use of an ongoing cycle of inquiry designed 
through this study can be shared with participants as a way to model this important leadership 
practice to move an organization further toward a compelling vision of success and 
achievement. 
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